Australian Hansard topic modelling analysis

pacman::p_load(stringr, tidyverse, tidytable, purrr, ggplot2)

all <- TRUE

if(all) {
  
  params <- "k40_var0.05"
  
  dtm_run_path <-
    "coal_output/dtm/run_21Sep//all_2a_min_freq_150_21_09_21/"
  
  
  docs_file <- "id_alliance_all_2a.csv"
  
  elec_period <- 1901:2020
  
  minimum_display_year <- 1900
  
} else {
  
  params <- "k25_var0.05"
  
  dtm_run_path <-
    "coal_output/dtm/run_21Sep//all_2a_last_20_years_min_freq_80_21_09_21/"
  
  docs_file <- "id_alliance_last20_2a.csv"
  
  elec_period <- 2000:2020
  
  minimum_display_year <- 2000
  
  
}


main_eras <- c(1935, 1963, 2005)

input_df %>% 
  count.(final_alliance)

input_df %>% 
  count.(final_alliance)

Raw number of docs by party

relative to parliamentary activity

https://www.aph.gov.au/-/media/02_Parliamentary_Business/22_Chamber_Documents/224_Statistics/House_of_Representatives/Statistics_Historical/legislation_statistics.pdf?la=en&hash=6ADCACEE5832882C1E8660A06563DC9200F6EC54

https://www.aph.gov.au/Parliamentary_Business/Bills_Legislation/Bills_Lists/Details_page?blsId=legislation%2Fbillslst%2Fbillslst_c203aa1c-1876-41a8-bc76-1de328bdb726

dividing the counts of paragraphs discussing coal by the number of bills passed in parliament that year as a proxy for parliamentary activity. The results are then normalised so that 1 corresponds to the largest amount of discussion by any party, which corresponds to the Greens in 2019. Parliament has arguably become more active over time in part due to an increase in the number of MPs and Senators and in part due to the increased requirements of a larger population and a more complex economy.

With those differences taken into account, the periods between 1940 and 1960 and post-2005 stand out, corresponding respectively to

largest amount of discussion

years above the 80% value: 1902, 1918, 1920, 1921, 1926, 1929, 1932, 1933, 1938, 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 1949, 1950, 1951, 1952, 1955, 1956, 1957, 1958, 1959, 1961, 1962, 1977, 1982, 1994, 2001, 2006, 2007, 2008, 2009, 2011, 2012, 2013, 2014, 2015, 2017, 2018, 2019

Bringing various elements together

filtering the minimal part of “other” in the period prior to 1975 and in general any topic before 1910 because the amounts of information are tiny and lead to wild fluctuations in proportion.

# From this article https://www.blueprintinstitute.org.au/polling_data
coal_mining_regions <- "Kalgoorlie|Parkes|Lyne|Hunter|calare|shortland|Flynn|Maranoa|gippsland|connor|capricornia"

combined_df <- input_df %>% 
  select.(doc_id, electorate, final_alliance, year) %>% 
  left_join.(select.(topic_distributions, -c(V1, year))) %>% 
  pivot_longer.(c(`0`:sym(topic_nums)), names_to = "topic_no", values_to = "score") %>% 
  mutate.(topic_no = as.integer(topic_no)) %>% 
  left_join.(topic_labels) %>% 
  left_join.(select(elections, year, era, legislature)) %>% 
  filter.(year > minimum_display_year, 
          !(final_alliance == "Other" & year < 1975)) %>% 
  mutate.(label = paste0(str_pad(topic_no, 2, "left", "0"), "_", label),
          mining_region = str_detect(electorate, regex(coal_mining_regions, ignore_case = T)))

Foci of each party relative to average

(Replicating MH’s Fig. 10)

Differences range: -3.218287 6.038361

Differences range: -9.190999 20.52203

Manually aggregated superlabels

TF-IDF ‘best superlabels’

best used to “confirm” that we’re not selecting the wrong thing.

MH-style supercategory

By individual topic and era

00_searborne transport01_national procurement02_electorate04_carbon price05_production and regions06_coal supply07_QLD and WA resources08_elections & policy09_budget10_committee on coal impacts11_natural disasters (GBR)12_fed vs states (JCB)13_trade partners (UK, JPN, CHN)14_renewable energy projects15_export16_research17_welfare18_big business (Adani)19_fed vs states (JCB)20_water and farming21_arbitration22_employee entitlements23_excise and duties24_nuclear power26_taxes27_coalminers28_commissions and committees29_other fuels30_employment31_production32_electricity generation33_Asian opportunities34_infrastructure35_bills and measures36_industrial relations37_iron and steel39_climate change